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DETAILED ACTION 

1 . Examiner acknowledges applicant's amendment filed on 7/1 0/2006. 

2. Claims 1-6,1 1 ,13,17 have been cancelled [7/10/2006]. 

3. Claims 7, 12. have been amended [7/10/2006]. 

4. Claims 1 8-29 have been added [7/1 0/2006]. 

Drawings 

5. The Drawings filed on 7/30/2003 are acceptable for examination purpose. 

Priority 

6. Applicant's claim for domestic priority under 35 U.S.C. 1 1 9(e) is acknowledged 
based on the provisional application 60/400,143 f\\ed on 7/31/2002; 60/400,213, filed on 
7/31/2002.. 

Information Disclosure Statement 



7. The information disclosure statement filed on 1 1/28/2003 is in compliance with 
the provisions of 37 CFR 1 .97, and has been considered and a copy is enclosed with 
previous Office Action mailed on 2/10/2006. 
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Interview: 

8. Applicant's Attorney Karl Kenna, Regd.No. 45,445 is thanked for the telephone 
interview on 13 September 2006. During that telephone interview Karl Kenna granted 
authorization to amend claims 7. 10. 12. 16. 18-22.24-28 . 

EXAMINER'S AMENDMENT 

9. An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1.312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview 
with Karl Kenna, Regd.No. 45,445 on 13 September 2006. 

The application has been amended as follows: 

In the Claims: Claims 7. 10. 12. 16. 18-22 and 24-28. all as shown below . 

1-6. (Canceled). 

7. (Currently Amended) A system for memory management comprising: 
a computer system including a virtual machine operating thereon; 
a memory space within said computer system and accessible by the virtual 

machine for the runtime storage and execution of applications; and, 
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a garbage collector that uses a reinforcement learning process to control the 
allocation of memory to applications within the memory space; 
wherein the garbage collector performs the steps of 

(a) measuring system-wide and application-specific parameters , and 
system events, to determine a current state of the memory soace , including the 
degree of fragmentation in the memory space , 

(b) performing an action to adjust the allocation of memory in the 
memory space, including garbage collecting the memory space, 

(c) calculating a reward value that indicates the success of the action 
on the memory space, including adding or subtracting preset values for specified 
actions of the garbage collector or conditions of the memory space. 

(d) storing information about the state, action and reward value for 
subsequent use by the garbage collector 

(e) subseguentlv measuring the system-wide and application-specific 
parameters, and system events, to determine a new state of the memory space. 

monitor i ng syst e m ev e nts that contr i but e to th e state, 

(f) retrieving and using the stored information about the state, action and 
reward value to determine an optimal action that the garbage collector should perform 
on the memory space to maximize a likely future reward value. 

(g) performing actions by the determined action by the garbage collector 
to adjust the allocation of memory. 
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(h) calculating a new reward value that indicates the success of the 
determined action on the memory space , and moacuring the systom w i d e and 
appl i cat i on sp e cific param e t e rs to d e t e rm i n e a n e w otato associat e d w i th tho reward. 
and, 

ill repeating the steps fd) throuoh (h) as n e c e ssary to control the 
allocation of memory to applications within the memory space. 

8. (Original) The system of claim 7 wherein the virtual machine is a Java Virtual 
Machine. 

9. (Original) the system of claim 7 wherein the reinforcement learning uses a 
temporal difference method. 

10. (Currently Amended) The system of claim 9 wherein the temporal difference 
method uses an on-policv on -li n e SARSA algorithm that uses the information about the 
state, action and reward value to first determine an optimal action that the aarbaoe 
collector should perform, and then updates the information about the new state, action 
and reward value, for subsequent use in determinino additional actions . 

11. (Canceled). 
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12. (Currently Amended) A method for memory management comprising the steps 
of: 

analyzing the memory space of a computer system or virtual machine; and, 
using a garbage collector together with a reinforcement learning process to 
control the allocation of memory to applications within the memory space 
wherein the garbage collector performs the steps of 

(a) measuring system-wide and application-specific parameters , and 
system events, to determine a current state of the memory space , including the degree 
of fragmentation in the memory space , 

(b) performing an action to adjust the allocation of memory in the memory 
space, including garbage collecting the memory space, 

(c) calculating a reward value that indicates the success of the action on 
the memorv space, including adding or subtracting preset values for specified actions of 
the garbage collector or conditions of the memory space, 

(d) storing information about the state, action and reward value for 
subseguent use by the garbage collector. 

(e) subseguentlv measuring the svstem-wide and application-specific 
parameters, and system events, to determine a new state of the memory space. 

mon i tor i ng syst e m ovonts that contr i but e to the ctato, 

(fi retrieving and using the stored information about the state, action and 

reward value to determine an optimal action that the garbage collector should perform 

on the memorv space to maximize a likely future reward value. 
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(g) performing actions by the determined action bv the garbage collector 
to adjust the allocation of memory, 

(h) calculating a new reward value that indicates the success of the 
determined action on the memory space , and moaour i ng th e syst e m w i do and 
app li cat i on specif i c paramet e rs to det e rm i n e a n e w state assoc i at e d with th e r e ward, 
and, 

(ii repeating the steps (d) through (h) as n e cessary to control the 
allocation of memory to applications within the memory space. 

13. (Canceled). 

14. (Original) The method of claim 12 wherein the virtual machine is a Java Virtual 
Machine. 

15. (Original) The method of claim 12 wherein the reinforcement learning uses a 
temporal difference method. 

16. (Currently Amended) The method of claim 15 wherein the temporal difference 
method uses an on-policv on l i n e SARSA algorithm that uses the information about the 
state, action and reward value to first determine an optimal action that the garbage 
collector should perform, and then updates the information about the new state, action 
and reward value, for subseguent use in determining additional actions . 
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17. (Canceled). 



Pages 



18. (Currently Amended) The system of claim 7, wherein the system-wide and 
application-specific parameters used to determine a the current state include the 
amount of memory allocated memory p e r by the system within a particular time unit. 

19. (Currently Amended) The system of claim 7, wherein the system-wide and 
application-specific parameters used to determine a current state include the amount of 
allocated memory the last time a decision to garbage collect was made. 

20. (Currently Amended) The system of claim 7, wherein the system-wide and 
application-specific parameters used to determine a current state include how much of 
the memory space is fragmented. 

21. (Currently Amended) The system of claim 7, wherein the system-wide and 
application-specific parameters used to determine a current state include any of the 
average size of new allocated objects, average age of allocated objects, or average 
amount of new allocated objects. 
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22. (Currently Amended) The system of claim 7, wherein the system events that 
contribute to the current state and for which preset reward values are associated 
include any of a var i abl e r e pr e s e nt i ng if whether a garbage collection was made during 
the last time step, a var i ab le r e pr e s e nting i f whether the system ran out of memory 
during the last time step, the amount of occupied memory before the garbage collection, 
the amount of occupied memory left after completed garbage collection, or the numb e r 
of situat i ons wh e r e a h e ap l ock n ee d e d to b e tak e n w hether the garbage collector made 
a heap lock during the last time step . 

23. (Previously Presented) The system of claim 7, wherein the actions by the 
garbage collector to adjust the allocation of memory include whether to garbage collect 
or not, whether to extend or compact the memory space, and by how much. 

24. (Currently Amended) The method of claim 12, wherein the system-wide and 
application-specific parameters used to determine a the current state include the 
amount of memory allocated m e mory p e r by the system within a particular time unit. 

25. (Currently Amended) The method of claim 12, wherein the system-wide and 
application-specific parameters used to determine a current state include the amount of 
allocated memory the last time a decision to garbage collect was made. 
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26. (Currently Amended) The method of claim 12. wherein the system-wide and 
application-specific parameters to used determine a current state include how much of 
the memory space is fragmented. 

27. (Currently Amended) The method of claim 12, wherein the system-wide and 
application-specific parameters used to determine a current state include any of the 
average size of new allocated objects, average age of allocated objects, or average 
amount of new allocated objects. 

28. (Currently Amended) The method of claim 12, wherein the system events that 
contribute to the current state and for which preset reward values are associated 
include any of a variab le r e pr e s e nting if whether a garbage collection was made during 
the last time step, a variab le repr e s e nt i ng if whether the system ran out of memory 
during the last time step, the amount of occupied memory before the garbage collection, 
the amount of occupied memory left after completed garbage collection, or tho number 
of s i tuat i ons wh e r e a heap l ock n e ed e d to be taken w hether the garbage collector made 
a heap lock during the last time step . 

29. (Previously Presented) The method of claim 12, wherein the actions by the 
garbage collector to adjust the allocation of memory include whether to garbage collect 
or not, whether to extend or compact the memory space, and by how much. 
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In the Title: 

An examiner's amendment to tlie record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1 .312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 

Pursuant to MPEP 606.01 the Title is changed to read: 

— SYSTEM AND METHOD FOR GARBAGE COLLECTION USES REINFORCEMENT 
LEARNING PROCESS TO PERFORM ADJUST THE ALLOCATION OF MEMORY 
SPACE, CALCULATING REWARD VALUE THAT INDICATES SUCCESS OF THE 
ACTION ON THE MEMORY SPACE AND MEASURING APPLICATION SPECIFIC 
PARAMTERS, SYSTEM EVENTS TO DETERMINE NEW STATE OF THE MEMORY 
SPACE - 
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Reasons for allowance 

Claims 7-10,12,14-16,18-29 are allowed 

The following is an examiner's statement of reasons for 
indication of allowable subject matter: The prior art of record does not disclose, 
make obvious, or othenA^ise suggest the structure of the applicant's memory 
management "(c) calculating a reward value that indicates the success of the action 
on the memory space, including adding or subtracting preset values for specified 
actions of the garbage collector or conditions of the memory space, 

(d) storing infonvation about the state, action and reward value for 
subsequent use by the garbage collector, 

(f) retrieving and using the stored infomiation about the state, action and reward 
value to determine an optimal action that the garbage collector should perfonv on the 
memory space to maximize a likely future reward value, 

(h) calculating a new reward value that indicates the success of the determined 
action on the memory space" in claim 7,12. 

These features, together with the other limitations of the independent claims 
are novel and non-obvious over the prior art of record. The dependent claims 8-10, 
14-16,18-29 being definite, enabled by the specification, and further limiting to the 
independent claims are also allowable. 
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Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance". 

if attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Alam, Hosain, T, can be reached on (571) 272-3978. The fax phone 
numbers for the organization where the application or proceeding is assigned is 
571-273-8300 Information regarding the status of an application may be obtained 
from the Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. 
Should you have questions on access to the Private PAIR system, contact the 
Electronic Business Center (EBC) at 866-217-9197 (toll-free) 

sc 

Patent Examiner. 
September 13, 2006. 



